Skip to content

Bugfix: Server-initiated room disconnect shutdown fix#205

Open
alan-george-lk wants to merge 15 commits into
mainfrom
alan/bugfix-room-shutdown
Open

Bugfix: Server-initiated room disconnect shutdown fix#205
alan-george-lk wants to merge 15 commits into
mainfrom
alan/bugfix-room-shutdown

Conversation

@alan-george-lk

@alan-george-lk alan-george-lk commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Fixes server-initiated room disconnect shutdown by routing explicit disconnect, destructor cleanup, server disconnect events, and EOS through a shared shutdown() helper, which by proxy cleans up event code
  • Ensure server Disconnected events remove the C++ FFI listener, stop subscriptions, clear room-owned state, and fire onDisconnected exactly once without re-sending an FFI disconnect

Before fix, when server deleted the room:

[2026-07-13 10:02:31.515] [livekit] [error] FfiClient listener threw: mutex lock failed: Invalid argument

Above observed on Mac, but crashed on Linux (user-reported). No longer after the fix.

Testing

  • Original issue reproduced via local binary (not committed), then fixed and confirmed no longer occurred
  • Add regression coverage for synthetic FFI server disconnects and local-SFU room deletion via lk --dev room delete
  • Consolidates emitFfiEvent into standalone header helper ffi_utils.h for use across two tests now

Ticket

BOT-464

@alan-george-lk alan-george-lk changed the title Bugfix: server-initiated room disconnect teardown fix Bugfix: Server-initiated room disconnect teardown fix Jul 13, 2026
@alan-george-lk alan-george-lk marked this pull request as ready for review July 13, 2026 22:36

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

Open in Devin Review

@xianshijing-lk xianshijing-lk left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, assuming you address those comments.

Comment thread include/livekit/room.h Outdated
Comment thread src/room.cpp
const bool has_room_state = connection_state_ != ConnectionState::Disconnected || listener_id_ != 0 ||
room_handle_ || local_participant_ || !remote_participants_.empty();
if (teardown_started_ || !has_room_state) {
return false;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we return true if nothing to do or it is being torndown ?

@alan-george-lk alan-george-lk Jul 14, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to changing this, right now our public API for disconnect() states:

@returns true if the graceful disconnect succeeds; false if the room was already disconnected (no-op) or the graceful disconnect fails.

Which this matches, false == noop. I get how that could be confusing, but I think the idea is true is when the disconnect actually happened

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is fair. We can keep the current behavior to avoid breaking changes.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW we have a public connectionState so apps can check the state before trying to disconnect

Comment thread src/tests/unit/test_room.cpp Outdated
alan-george-lk and others added 2 commits July 13, 2026 21:53
Align the shared cleanup helper with LocalParticipant/FfiClient naming,
keep disconnect() returning false for no-ops per its documented contract,
and cover Room reuse after server-initiated shutdown.

Co-authored-by: Cursor <cursoragent@cursor.com>
@alan-george-lk alan-george-lk changed the title Bugfix: Server-initiated room disconnect teardown fix Bugfix: Server-initiated room disconnect shutdown fix Jul 14, 2026
devin-ai-integration[bot]

This comment was marked as resolved.

@xianshijing-lk

Copy link
Copy Markdown
Collaborator

lgtm, thanks

devin-ai-integration[bot]

This comment was marked as resolved.

Comment thread src/room.cpp
}

// old_* state is destroyed here when going out of scope
(void)shutdown(false, DisconnectReason::Unknown, false);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is EOS not a DisconnectReason?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: Confirm the flow is this on server disconnect:

  • Server disconnect event (do shutdown)
  • EOS (don't need shutdown)

Verify that the EOS is double-calling shutdown right now as written (after server close)

@stephen-derosa stephen-derosa left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice lgtm!

@alan-george-lk alan-george-lk force-pushed the alan/bugfix-room-shutdown branch from 3cbddb6 to a2e6a9e Compare July 14, 2026 19:12
devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

View 5 additional findings in Devin Review.

Open in Devin Review

Comment thread src/room.cpp
Comment on lines +1196 to 1209
bool should_notify = false;
{
const std::scoped_lock<std::mutex> guard(lock_);
already_disconnected = (connection_state_ == ConnectionState::Disconnected);
connection_state_ = ConnectionState::Disconnected;
}
if (already_disconnected) {
break;
}
DisconnectedEvent ev;
ev.reason = toDisconnectReason(re.disconnected().reason());
if (delegate_snapshot) {
// Local shutdown marks the state before awaiting the FFI response
// and notifies the delegate itself. Suppress that duplicate while
// passing server-initiated disconnects through unchanged.
should_notify = connection_state_ != ConnectionState::Disconnected;
}
if (should_notify && delegate_snapshot) {
DisconnectedEvent ev;
ev.reason = toDisconnectReason(re.disconnected().reason());
delegate_snapshot->onDisconnected(*this, ev);
}
break;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Server disconnect notification can fire twice when a user disconnect races with a server disconnect event

The disconnect notification is decided (should_notify = true at src/room.cpp:1202) without updating the room's connection state, so a concurrent user-initiated disconnect on another thread can also pass its own state check and fire the same notification, resulting in the delegate receiving two disconnect callbacks instead of one.

Impact: The application's disconnect handler runs twice with different reasons, which can cause double-cleanup, incorrect state transitions, or crashes in user code that assumes exactly-one semantics.

Race window between kDisconnected handler and user disconnect()

The old code at the base commit always set connection_state_ = Disconnected inside the kDisconnected handler under the lock. This created mutual exclusion: whichever path (FFI event or user disconnect()) set the state first would prevent the other from firing the delegate.

The new code at src/room.cpp:1196-1209 only reads the state but never writes it:

bool should_notify = false;
{
    const std::scoped_lock<std::mutex> guard(lock_);
    should_notify = connection_state_ != ConnectionState::Disconnected;
    // ← state is NOT updated here
}
if (should_notify && delegate_snapshot) {
    delegate_snapshot->onDisconnected(*this, ev);
}

Race sequence:

  1. FFI thread: kDisconnected handler acquires lock, reads Connected, sets should_notify = true, releases lock.
  2. User thread: disconnect()shutdown() acquires lock, also reads Connected (unchanged!), moves out all state, sets Disconnected, releases lock, calls delegate->onDisconnected(ClientInitiated).
  3. FFI thread: resumes and calls delegate->onDisconnected(RoomDeleted) because should_notify was already captured as true.

Result: delegate receives two onDisconnected calls. The integration test UserDisconnect explicitly asserts delegate.count.load() == 1, so this race can cause test failures and violates the documented contract.

Prompt for agents
In src/room.cpp, the kDisconnected event handler (around line 1196-1209) reads connection_state_ under the lock to decide whether to notify the delegate, but does not update the state. This creates a race window where a concurrent user-initiated disconnect() on another thread can also pass its state check and fire the delegate, resulting in two onDisconnected calls.

The fix should restore the old behavior of setting connection_state_ = ConnectionState::Disconnected inside the kDisconnected handler's locked section, matching what the base commit did. This ensures mutual exclusion: whichever path (the FFI kDisconnected event or the user's disconnect() call to shutdown()) sets the state first will prevent the other from firing the delegate.

Specifically, in the kDisconnected case block, after computing should_notify, add: connection_state_ = ConnectionState::Disconnected; inside the locked section. This way shutdown() on another thread will see the state as already Disconnected and return false (no-op), preventing the double notification.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants